1 link tagged with all of: data engineering + streaming architecture
Click any tag below to further narrow down your results
Links
WHOOP built a CDC pipeline that automatically syncs Postgres schema changes to Snowflake without manual intervention, by splitting the problem into two layers—a cheap append-only Bronze layer for raw events and a periodic Silver layer for materialized replicas—rather than doing continuous upserts like most systems.
- The old system ran hundreds of expensive Spark streaming jobs doing continuous upserts every 30 minutes; the new design cuts this to one Flink job per database using dynamic routing, eliminating the constant reconciliation overhead.
- Schema changes now propagate automatically: adding a column to Postgres flows through Debezium to Avro to Iceberg without tickets or manual backfills, because the publication definition lives in the service team's own repo alongside their Liquibase migrations.
- Splitting streaming (Bronze, append-only, immediate) from upserting (Silver, every 4 hours, materialized) lets consumers choose their tradeoff—most don't need real-time state and can query the cheaper, simpler Bronze layer.